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This  report  describes  the  work  done  by  Decisions  and 
Designs,  Incorporated,  in  collaboration  with  the  United 
States  Marine  Corps,  on  the  development  of  a methodology  to 
improve  the  1979  Program  Objectives  Memorandum  (POM)  process. 
The  generic  approach  chosen  was  cost-benefit  analysis.  The 
goal  of  cost-benefit  analysis  is  to  identify,  for  subsequent 
funding,  those  programs  that  optimize  the  total  benefit  to  a 
decision  maker  within  his  budget  constraints.  The  identifi- 
cation of  optimal  procurements  is  based  upon  determination 
of  the  "true"  costs  and  benefits  of  each  program.  "True"  or 
absolute  costs  and  benefits  are  often  very  difficult  to 
determine  because  subjective  estimates  must  be  made,  and 
because  some  of  the  less-than-obvious  costs  and/or  benefits 
may  be  overlooked.  Fortunately,  for  this  application  esti- 
mates of  the  absolute  costs  and  benefits  were  not  required, 
only  estimates  of  the  relative  costs  and  benefits. 


The  programs  involved  in  this  analysis  were  all  future 
capability  items  in  the  procurement  appropriation.  DDI's 
efforts  were  focused  on  obtaining  good  subjective  estimates 
of  the  relative  benefits  for  each  of  these  programs.  Psy- 
chologists and  decision  analysts  have  observed  that  the  best 
way  to  obtain  accurate  quantifications  of  this  sort  is  to 
use  paired  comparisons,  that  is,  to  ask  the  expert  to  make  a 
series  of  choices  between  two  packages  of  progreuns,  each 
choice  having  certain  implications  about  the  benefits  of  the 
programs. 


The  elicitation  procedure  used  for  obtaining  the  benefit 
estimates  was  an  iterative  one.  The  USMC  POM  programs  were 
divided  into  eight  categories,  each  category  having  a sponsor  'ON 
who  was  especially  knowledgeable  about  his  subset  of  programs. 
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The  elicitation  procedure  began  with  an  ordinal  listing  of 
each  sponsor's  programs  by  benefit.  Once  an  initial  ratio 
scale  was  defined  over  these  programs,  the  sponsor's  responses 
to  choices  between  numerous  sets  of  programs  generated  the 
feedback  necessary  to  modify  and  improve  this  ratio  scale  of 
benefits.  After  three  or  four  iterations  of  this  elicitation 
procedure,  each  sponsor  was  satisfied  that  his  benefit  scale 
reflected  his  preferences. 

The  next  step  was  to  convene  a group  of  "honest-brokers," 
officers  from  the  Headquarters  Staff  sections,  which  do  not 
sponsor  any  programs.  This  group  was  asked  to  provide  a 
benefit  scale  for  a small  subset  of  all  the  programs;  the 
subset  included  one  program  from  each  sponsor's  list.  After 
the  "honest-brokers"  had  been  completely  educated  about  the 
uses  of  the  programs  being  considered,  the  same  elicitation 
procedure — that  described  above — was  used  to  obtain  their 
benefit  scale.  This  scale  provided  all  of  the  information 
needed  to  collapse  the  eight  distinct  sponsor  benefit  scales 
into  one. 

This  cross-sponsor  elicitation  of  benefits  provided  a 
proper  scoring  rule.  The  meaping  of  the  proper  scoring  rule 
was  explained  to  each  sponsor  in  the  course  of  eliciting  his 
benefits:  If  the  sponsor  ranked  the  lower-benefit  procure- 

ments too  high  on  his  benefit  scale,  the  cross-sponsor 
weighting  would  drag  all  of  his  high-benefit  procurements 
down  in  relation  to  those  of  the  other  sponsors.  A proper 
scoring  rule  for  subjective  elicitations  of  this  sort  thus 
motivates  the  sponsors  to  provide  their  "true"  beliefs  in 
order  to  maximize  their  final  position  in  the  resource 
allocation.  Finally,  each  sponsor  and  the  "honest-brokers" 
wrote  concise  justifications  of  their  respective  benefit 
scales.  These  justifications  were  used  for  briefing  the  POM 
recommendations . 
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There  were  several  advantages  to  the  systematic,  exhaus- 
tive analysis  that  generated  the  benefits  and  produced  the 
cost-benefit  ordering  of  procurements.  First,  the  justifica- 
tion for  the  POM  recommendations  was  right  at  the  fingertips 
of  the  POM  working  group  when  the  analysis  was  completed. 
Second,  the  "hands-on"  availability  of  interactive  computer 
software  on  an  IBM  5100  minicomputer  allowed  the  POM  working 
group  charged  with  preparing  the  recommendations  great 
flexibility  for  evaluating  the  sensitivity  of  benefits  and 
determining  the  implications  of  funding  progreims  that  were 
not  originally  included.  Third,  the  cost-benefit  approach 
facilitated  the  rapid  identification  and  solution  of  important 
decision  areas.  Fourth,  the  analysis  resulted  in  a general 
but  more  thorough  education  of  the  POM  working  group  concerning 
the  benefits  of  all  the  programs.  Finally,  future  POM  efforts 
are  going  to  benefit  because  of  the  corporate  memory  and 
transferability  of  this  process. 
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1.0  INTRODUCTION  AND  BACKGROUND 


1.1  Introduction 

This  report  describes  the  work  done  by  Decisions  and 
Designs,  Incorporated  (DDI) , in  collaboration  with  the 
United  States  Marine  Corps  (USMC) , on  the  development  of 
methodology  to  improve  the  1979  Program  Objectives  Memoran- 
dum (POM)  process.  The  first  of  the  four  sections  in  the 
report  describes  the  background  of  the  specific  problem. 

The  technical  work  done  by  DDI  for  the  POM  development  is 
presented  next.  The  third  section  describes  the  benefits  of 
the  research  effort  to  the  Marine  Corps.  The  fourth  section 
discusses  important  future  work  and  also  serves  as  a conclu- 
sion. 

The  purpose  of  this  application  project  was  to  develop 
an  improved  methodology  to  help  the  Marine  Corps  prepare 
part  of  its  portion  of  the  Department  of  the  Navy  Program 
Objectives  Memorandum  (POM)  for  the  1979-1983  time  period. 

The  1979  program  year  was  most  crucial  since  it  would  almost 
immediately  be  translated  into  the  1979  budget  and  be  subject 
to  defense  before  Congress.  The  appropriation  dollars 
available  for  manpower,  operations  and  maintenance,  reserves, 
and  part  of  procurement  are,  to  a large  degree,  relatively 
fixed  and  constitute  over  90%  of  a typical  budget,  as  illus- 
trated in  Figure  1-1. 

The  most  flexible  and  .difficult  decisions  concerning 
the  programming  of  funds  are  found  in  the  procurement  area, 
which,  for  programming  purposes,  is  divided  into  ammunition 
and  both  current  and  future  capabilities.  Our  analysis 
centered  around  the  procurement  of  future  capability  items. 
The  projected  funding  profile  in  this  area  for  1979  through 
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IL,- — 


FY-78 

TOTAL  TOA  - $3,117  (SM)* 


$1,924  $637  *96  *460 

( 62%  + 20%  + 3%  ♦ 15%  ' 100%  ) 


'Does  not  include  stock  fund  ($2.0M) 

Figure  1-1 
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1983  is  illustrated  in  Table  1-1  along  with  the  costs  of 
validated  requirements.  As  shown,  there  are  some  years  in 
which  the  requirements  exceed  the  total  obligational  autho- 
rity (TOA)  available  by  more  than  200%. 

1 . 2 Background 

Figure  1-2  lists  the  eight  program  sponsors  at  Head- 
quarters Marine  Corps,  each  of  whom  sponsors  various  future 
capability  items;  each  item  has  only  one  sponsor.  In  the 
past,  these  sponsors  have  entered  the  "smoke-filled"  room 
with  an  ordinal  listing  of  their  own  procurement  items. 

Each  would  then  agree  to  give  up  their  least  desired  items 
until  the  total  costs  for  the  five  years  matched  the  pro- 
jected TOA.  This  was  clearly  a suboptimal  and  non-systematic 
procedure,  adapted  from  a Navy  POM  procedure  with  the  same 
traits.  The  Marine  Corps  was  unhappy  with  it  because  it  was 
difficult  to  determine  which  items  would  be  eliminated  if  a 
last-minute  insertion  were  made,  how  adjustments  would  be 
made  if  the  costs  were  to  change  at  the  last  minute,  or  what 
the  effects  of  other  last-minute  changes  might  be. 

DDI  analysts  pointed  out  that  this  procedure  determined 
which  procurements  should  be  made  by  a benefit-only  criterion 
within  each  sponsor  and  that  it  permitted  no  clear  trade-offs  of 
the  benefits  among  sponsors.  A simple  cost-benefit  analysis, 
such  as  the  one  illustrated  in  Figure  1-3,  was  clearly  more 
optimal.  This  figure  shows  the  hypothetical  purchase  of 
items  based  upon  both  benefit-only  and  cost-benefit  criteria. 

The  horizontal  axis  represents  accumulated  dollars  as  items 
are  purchased,  the  vertical  axis  the  accumulated  benefit. 
Whenever  the  fiscal  constraint  is  as  restrictive  as  the  one 
the  Marine  Corps  faced  in  the  development  of  POM  79,  the 
cost-benefit  criterion  yields  greater  benefits  than  the 
benefit-only  criterion. 


i 

J 


3 


PMC 
(SM  78) 


Future  Capabilities: 

FY  79 

FY80 

FY  81 

FY82 

FY83 

TO  A AVAILABLE 

106 

75 

106 

215 

271 

COST  OF  VALIDATED 

181 

286 

325 

306 

253 

REQUIREMENTS 

Table  1-1 


• TRAINING 

• INSTALLATION  AND  LOGISTICS 

• AVIATION 

• RESERVES 

• TELECOMMUNICATIONS 

• INTELLIGENCE 

• OPERATIONS 

• 

Figure  T2 

MARINE  POM  SPONSORS 
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COST-BENEFIT  VS.  BENEFIT-ONLY  CRITERIA 


2.0  TECHNICAL  APPROACH 
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2.1  The  Cost-Benefit  Criterion 

The  well-khown  cost-benefit  criterion  for  choosing 
programs  attempts  to  optimize  the  total  benefit  to  the 
decision  maker  for  any  level  of  expenditure.  This  could 
mean  the  purchase  of  a number  of  options  that  are  not  at  the 
top  of  the  benefit  scale  because  their  costs  are  substan- 
tially less  than  options  with  higher  benefits. 

Determining  the  "true"  costs  and  benefits  of  each 
alternative  can  be  a very  difficult  task.  Take,  for  instance, 
the  government  directive  that  all  agencies  are  to  use  the 
"life-cycle"  costs  of  a potential  procurement  in  deciding 
whether  to  purchase  it  or  not.  There  are  obvious  and  not- 
so-obvious  elements  of  life-cycle  cost:  The  first  obvious 
element  is,  of  course,  the  cost  of  the  procurement  itself, 
which  typically  includes  the  cost  of  supporting  equipment, 
e.g.,  all  guns  require  bullets.  Training  costs  for  the 
procured  system  should  also  be  included.  Other  less  obvious 
costs  typically  considered  include  research  and  development 
costs  and  operating  costs.  But  where  does  one  stop?  Should 
the  costs  of  naval  hospitals  and  retirement  benefits  be 
proportioned  to  the  life-cycle  costs  of  submarines  and 
aircraft  carriers? 

Still  another  difficult  aspect  of  life-cycle  costs  is 
their  uncertainty.  All  of  these  costs  are  to  be  incurred  in 
the  future,  and  thus  they  contain  all  of  the  uncertainty 
associated  with  projections  of  the  future. 

The  next  two  sections  discuss  the  procedures  DDI  ana- 
lysts used  to  assess  the  benefits  from  each  sponsor  for  his 
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procurements  and  from  an  "honest-broker"  group  for  the 
procurements  of  all  the  sponsors.  These  assessments  were 
facilitated  by  specially-programmed  software  for  the  IBM 
5100  mini-computer. 

2 . 2 The  Elicitation  of  Benefits  Within  a Sponsor's  Proposed 
Procurements 


Psychologists  and  decision  analysts  have  observed  that 
the  best  way  to  get  reliable  estimates  of  potential  benefit 
is  to  use  paired  comparisons,  that  is,  to  ask  the  expert  to 
make  a succession  of  choices  between  two  packages  until  a 
point  of  indifference  is  reached.  The  conventional  approach 
is  to  use  money  as  the  numeraire  attribute  to  establish 
indifference  between  two  bundles,  each  with  several  attri- 
butes. In  that  approach,  once  the  expert  has  established  an 
ordinal  preference  for  bundle  A over  bundle  B,  say,  he  is 
asked  if  he  prefers  A or  B plus  X dollars.  The  value  of  X 
is  adjusted  until  a point  of  indifference  is  established. 

The  relation  between  B and  the  status  quo,  neither  A nor  B, 
is  then  established  by  the  same  procedure.  This  method 
works  very  well  in  the  business  community,  but  since  it  has 
not  fared  very  well  with  decision  makers  in  the  government, 
DDI  analysts  developed  a method  that  did  not  require  the  use 
of  money  as  the  numeraire  attribute. 


2.2.1  Development  of  the  elicitation  procedure  - The 
elicitation  procedure  devised  for  this  application  required 
that  a sponsor's  procurements  first  be  ordinally  ranked  by 
their  total  benefits  and  then  assigned  directly  to  a ratio 
scale  for  individual  benefits.  The  resulting  scale  was  not 
too  reliable,  but  it  could  be  used  with  the  cost  data  to 
compute  cost-benefit  ratios  and  to  rank  the  procurements  in 
the  order  in  which  they  would  be  bought.  Then,  two  curves 
were  plotted  in  terms  of  cumulative  benefit  versus  cumulative 
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cost,  one  resulting  from  the  incremental  buys  by  the  benefit- 
only  criterion,  the  other  using  the  cost-benefit  criterion. 

An  example  of  these  curves  are  shown  in  Figure  2-1. 

The  cost-benefit  criterion  produces  a curve  that 
is  always  above  or  equal  to  the  benefit-only  curve.  Note 
that  the  slopes  of  the  straight  lines  in  the  cost-benefit 
curve  are  monotonically  decreasing  as  the  curve  goes  from 
left  to  right,  whereas  the  vertical  jumps  in  the  benefit- 
only  curve  are  monotonically  decreasing  as  each  succeeding 
procurement  is  made. 

The  curves  illustrate  which  procurements  would 
be  made  for  numerous  fiscal  constraints,  as  represented  by 
the  numerical  scale  on  the  abscissa.  Here,  we  assume  the 
procurements  are  divisible,  so  that  procurement  of  partial 
programs  is  meaningful. 

These  curves  also  provide  the  sponsor  with  a 
procedure  for  checking  the  accuracy  of  the  benefit  scale  he 
has  specified.  This  is  done  by  using  both  the  cost-benefit 
and  benefit-only  criteria  to  select  two  groups  of  procurements 
that  have  roughly  the  seune  cost.  An  example  of  such  a pair 
is  represented  by  the  two  groups  to  the  left  of  the  dotted 
line  in  Figure  2-1.  The  sponsor  must  decide  if  he  prefers 
the  procurements  on  the  cost-benefit  curve  to  the  one  on  the 
benefit-only  curve  and,  if  so,  by  roughly  how  much.  If  not, 
the  benefit  scale  can  be  adjusted  to  reflect  his  preferences. 
There  are  other  equal-cost  lines  that  can  be  drawn  to  check 
the  benefit  scale.  This  procedure  was  used  at  the  global 
level  of  benefit  judgments  but  could  also  be  used  at  lower 
levels  if  the  sponsor  were  more  comfortable  making  these 
judgments. 


9 


BENEFIT 


0 50  100  150  200  250  300 


COST  ($M) 

Figure  2-1 
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The  foregoing  elicitation  process  dictated  the 
following  iterative  procedure:  First,  a benefit  scale  is 
postulated,  and  a figure  similar  to  Figure  2-1  is  generated. 
Using  this  figure,  the  sponsor  checks  for  inconsistencies  at 
numerous  levels  of  funding;  then  the  benefits  of  the  procure- 
ments are  adjusted  until  the  inconsistencies  are  resolved. 

A new  figure  is  generated,  and  the  process  is  repeated  until 
no  inconsistencies  can  be  found. 

A large  number  of  alternative  consistency  checks 
can  be  made  with  this  benefit  scale  since  it  must  be  a ratio 
scale.  (Note  that  zero  corresponds  to  a program  with  no 
benefit.)  Many  of  these  consistency  checks  were  examined 
before  accepting  the  scale.  The  application  of  this  tech- 
nique produced  the  stimulus  needed  to  force  the  sponsor  to 
insert  realistic  numerical  differences  between  the  appropriate 
procurements  on  his  benefit  scale.  The  resulting  benefit 
scale  was  far  superior  to  any  generated  by  direct  scaling 
techniques. 

2.2.2  Illustration  of  the  elicitation  procedure  - The 
following  example  is  a useful  illustration  of  this  proce- 
dure. Suppose  there  are  ten  possible  procurements,  designated 
A through  J,  proposed  by  the  sponsor.  These  are  listed  in 
Table  2-1,  along  with  an  initial  benefit  scale,  total  cost, 
and  benefit/cost  ratio.  The  order  in  which  these  procure- 
ments would  be  made  according  to  the  cost-benefit  criterion 
is 


>J,  I,  D,  A,  F,  E,  G,  H,  G,  C. 

Figure  2-2  illustrates  the  difference  in  benefit  between  the 
cost-benefit  and  benefit-only  criteria  for  all  levels  of 
cost.  Note  that  using  the  cost-benefit  criterion  with  these 
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INITIAL 

PROCUREMENTS  BENEFITS 

TOTAL 

COSTS 

BENEFIT 

COST 

A 

100 

16 

6.2 

B 

99 

36 

2.8 

C 

95 

56 

1.7 

D 

90 

9 

10.0 

E 

87 

30 

2.9 

F 

83 

20 

4.2 

G 

70 

35 

2.0 

H 

70 

26 

2.7 

1 

60 

2 

30.0 

J 

55 

1 

55.0 

Table  2-1 

PROCUREMENTS  - INITIAL  COMPARISONS 
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benefit  numbers  is  almost  equivalent  to  ordering  the  pro- 
curements by  cost  in  descending  order. 


The  second  iteration  of  this  process  begins  by 
comparing  procurements  J,  I,  and  D with  procurement  A,  The 
package  J,  I,  and  D costs  nearly  as  much  as  procurement  A 
but  should  be  twice  as  beneficial  as  A.  However,  when  asked 
which  he  preferred,  the  sponsor  said  A had  more  benefit  than 
J,  I,  and  D.  So  A's  benefit  was  adjusted  to  250  to  reflect 
the  strength  of  his  preference. 

Next,  note  that  A and  B are  nearly  equivalent  to 
J,  I,  D,  A,  and  F in  cost.  Since  A is  common  to  both  packages 
and  there  are  no  interdependencies  between  the  procurements, 

B can  be  compared  to  J,  I,  D,  and  F.  In  this  case,  J,  I,  D, 
and  F were  strongly  preferred,  and  the  sponsor  felt  B was 
equivalent  to  J,  I,  and  D.  So  B's  benefit  was  raised  to 
215. 

The  next  two  equal-cost  packages  are  A,  B,  C, 
and  D,  and  J,  I,  D,  A,  F,  E,  and  B.  Since  A,  B,  and  D are 
common  to  both,  they  can  be  deleted;  therefore,  the  sponsor 
was  asked  to  compare  C to  J,  I,  F,  and  E.  He  felt  that  C 
was  equivalent  to  J,  I,  and  F or  had  a benefit  of  205.  The 
new  benefit  numbers  implied  that  B and  C are  more  beneficial 
than  A,  and  the  sponsor  verified  that  this  was  the  case. 

This  exhausted  the  useful  comparisons  between 
the  cost-benefit  and  the  benefit-only  criteria  in  Figure  2-2. 

A sizable  gap  in  benefit  between  A and  B had  been  identified, 
suggesting  that  several  other  comparisons  should  be  made  to 
expose  similar  gaps.  The  sponsor  felt  that  C and  D were 
preferred  to  either  B or  A.  However,  since  the  benefits  of 
D and  E were  considered  equivalent  to  C,  ten  units  were 
added  to  the  preliminary  values  of  D and  E.  This  required 
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that  B's  benefit  value  be  raised  to  225.  The  decision  maker 
also  felt  that  E and  F were  more  beneficial  than  D but  less 
beneficial  than  C.  So,  the  value  of  F was  kept  at  83.  The 
original  numbers  for  G,  H,  I,  and  J were  thought  to  be  rep- 
resentative. 

The  new  benefits  are  displayed  in  Table  2-2. 

The  third  column  gives  the  benefits  assigned  in  the  above 
process,  and  the  fourth  column  displays  the  benefits  norma- 
lized to  a zero-to-lCO  scale.  The  last  column  shows  the 
benefit-to-cost  ratios.  The  new  order  of  cost-benefit  buys, 
which  is  substantially  different  than  the  first,  is 

iJ,  1,  A,  D,  B,  F,  C,  E,  H,  G. 

In  Figure  2-3,  the  cumulative  benefits  of  these  procurements 
are  compared  to  the  cumulative  benefits  resulting  from  the 
benefit-only  criterion.  Now  benefit  and  cost  are  equal 
factors  in  the  cost-benefit  ordering  of  procurements. 

At  this  point,  it  was  necessary  for  the  sponsor 
to  explain  to  his  experts  and  superiors  the  choices  he  had 
to  make  in  deriving  this  new  benefit  scale.  In  doing  so,  he 
uncovered  additional  factors  that  led  him  to  move  procure- 
ment G up  the  benefit  ladder  to  a position  between  C and  D. 
Through  this  iterative,  comparative  process,  a final  benefit 
scale  was  constructed.  This  normalized  scale  is  presented 
in  Table  2-3. 

The  final  order  of  cost-benefit  buys  is 
J i I,  D,  A,  B,  G,  F,  C,  i.-,  H. 

This  process  helped  the  sponsor  to  develop  substantive  ration- 
ale for  supporting  the  final  benefit  scale.  Since  the  sponsor 
had  worked  closely  with  both  his  superiors  and  experts  during 
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PROCUREMENTS 

COSTS 

(S) 

INTERMEDIATE 

BENEFITS 

NORMALIZED 

BENEFITS 

BENEFIT 

COST 

A 

16 

250 

100 

6.2 

B 

36 

225 

90 

2.5 

C 

56 

205 

82 

1.5 

D 

9 

105 

42 

4.7 

E 

30 

100 

40 

1.3 

F 

20 

83 

33 

1.6 

G 

35 

70 

28 

0.8 

H 

26 

70 

28 

1.1 

1 

2 

60 

24 

12.0 

J 

1 

55 

22 

22.0 

Table  2-2 
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PROCUREMENTS 

COSTS 

(S) 

ORIGINAL 

BENEFITS 

FINAL 

(NORMALIZED) 

BENEFITS 

BENEFIT 

COST 

A 

16 

100 

100 

6.2 

B 

36 

99 

83 

2.3 

C 

56 

95 

80 

1.4 

G 

35 

70 

72 

2.1 

D 

9 

90 

58 

6.4 

E 

30 

87 

37 

1.2 

F 

20 

83 

30 

1.5 

H 

26 

70 

19 

0.7 

1 

2 

60 

15 

7.5 

J 

1 

55 

8 

8.0 

Table  2-3 

PROCUREMENTS  - FINAL  COMPARISONS 
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this  process,  the  scale  and  rationale  had  their  concurrence. 
Figure  2-4  shows  the  final  differences  between  buying  with 
the  cost-benefit  criterion  versus  the  benefit-only  criterion. 
For  a fiscal  constraint  of  $100,  the  cost-benefit  criterion 
provides  68%  of  the  possible  benefit,  which  is  a 33%  increase 
over  the  benefit-only  criterion. 

2 . 3 The  Elicitation  of  Benefits  Across  Sponsors'  Proposed 
Procurements 


Prior  to  the  introduction  of  the  cost-benefit  approach, 
trade-offs  across  sponsors  were  made  in  an  ad  hoc  manner. 

The  use  of  this  ad  hoc  method  was  dictated  largely  by  the 
difficulty  of  comparing  such  things  as  howitzers,  command 
and  control  systems,  reserve  procurements,  and  sophisticated 
communication  and  radar  equipment.  (The  essence  of  the  POM 
79  problem  is  illustrated  in  Figure  2-5.)  Nevertheless, 
because  of  the  diversity  of  programs  and  the  limited  amount 
of  funds,  it  is  important  that  these  trade-offs  among  sponsors 
be  made  explicit  so  that  rational  procurement  decisions  can 
be  made.  For  example,  one  of  the  conclusions  of  the  trade- 
off procedure  used  in  this  analysis  was  that  the  logistics 
area,  because  of  the  acquisition  of  sophisticated  equipment 
in  the  other  sponsors'  areas,  was  relatively  more  important 
than  traditionally  thought. 

2.3.1  Development  of  the  "honest-broker"  elicitation 
procedure  - The  eight  different  sponsor  benefit  scales  were 
collapsed  into  one  benefit  scale  by  using  an  "honest-broker" 
group  of  Marines  from  Headquarters  staff  sections,  which  do 
not  sponsor  any  programs.  The  "honest-broker"  group  repre- 
sented a variety  of  military  occupational  skills;  each 
member  was  given  comprehensive  information  about  items  to  be 
compared  and  was  asked  to  study  the  material  in  detail.  To 
further  educate  this  "honest-broker"  group,  a question-and- 
answer  session  was  held  with  sponsor  participation.  Here, 
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the  "honest-broker"  group  was  able  to  question  the  sponsors 
in  detail  until  they  felt  they  understood  each  of  the  items. 

First,  three  distinct  packages  of  procurements 
were  formed.  Each  package  had  one  procurement  item  from 
each  sponsor's  list.  One  package  contained  the  most  bene- 
ficial procurements  of  each  sponsor.  Another  contained  pro- 
curements that  needed  1979  funding  since  this  was  the  most 
important  year.  The  two  packages  containing  procurements 
' that  were  not  at  the  top  of  the  sponsors'  lists  served  as 

I proper  scoring  rules.  That  is,  if  the  sponsor  ranked  these 

I lower-benefit  procurements  too  high  on  his  benefit  scale, 

I 

the  cross-sponsor  weighting  would  drag  all  of  his  high- 
benefit  procurements  down  in  relation  to  those  of  the  other 
sponsors. 

This  elicitation  process  could  have  been  accom- 
plished with  only  one  such  package,  but  three  packages  were 
used  in  order  to  compensate  for  some  expected  inconsistencies 
in  the  cross-sponsor  judgments.  Inconsistencies  were  antici- 
pated because  the  "honest-broker"  group  was  not  as  familiar 
with  the  procurements  as  the  sponsors  were  and  because  these 
cross-sponsor  judgments  were  much  more  difficult  to  make. 

The  three  packages  of  procurements  highlighted  these  incon- 
sistencies so  that  intelligent  discussion  could  resolve 
them. 

2.3.2  Illustration  of  the  "honest-broker"  elicitation 
I procedure  - As  an  illustration  of  the  cross-sponsor  benefit 

scaling,  consider  the  following  two-sponsor  example.  Each 
sponsor,  A and  B,  has  four  procurements  and  has  assigned 
' benefits  as  shown  in  Figure  2-6.  The  "honest-broker"  group 

is  asked  to  compare  A-2  and  B-3,  and  decides  that  B-3  is 
twice  as  beneficial  as  A-2  (Figure  2-7).  (In  practice, 
there  were  eight  procurements  in  each  package , and  the 
I iterative  assessment  procedure  described  in  Section  2.2  was 

I 

( 
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used.)  This  assessment  then  provides  enough  information  to 
put  the  procurements  of  both  sponsors  on  one  scale  as  shown 
in  Figure  2-8. 


For  this  cross-sponsor  assessment  of  benefits, 
it  was  assumed  that  the  sponsor's  benefit  rankings  of  his 
own  programs  were  valid  and  could  not  be  changed.  This 
assumption  was  made  in  the  belief  that  the  sponsors  had  more 
intimate  knowledge  about  their  programs  than  the  "honest- 
broker"  group  and  that  it  was  easier  to  compare  those  pro- 
curements in  one  sponsor's  area  than  in  a set  of  cross- 
sponsor procurements.  After  the  "honest  brokers"  provided 
benefit  scales  for  the  three  distinct  packages,  several  in- 
consistencies were  noted.  All  but  one  were  very  minor  and 
easily  resolved.  The  most  serious  one  involved  a discrepancy 
between  the  results  of  two  of  the  cross-sponsor  assessments 
and  the  results  of  the  third  assessment.  A lengthy  discussion 
of  the  reasons  behind  the  inconsistency  finally  led  to  a 
resolution. 

2 . 4 Implications  of  the  Cost-Benefit  Analysis 

As  a comparative  exercise,  and  on  the  assumption  that 
cost  data  and  benefit  estimates  were  sufficiently  accurate, 

DDI  analysts  generated  the  final  accumulated-benef it  versus 
accvimulated-cost  curves  by  using  the  cost-benefit  and  benefit- 
only  criteria  for  five  of  the  eight  sponsors,  based  on  the 
best  cost  information  available.  These  curves  are  presented 
in  Figures  2-9  through  2-13.  The  other  three  sponsors  had 
only  one  procurement  item  each.  The  curves  for  the  Installa- 
tion and  Logistics  and  the  Operations  sponsors  were  close 
together;  however,  the  differences  in  benefit  at  given 
levels  of  cost  are  deceptively  large  for  curves  with  these 
slopes.  There  are  significant  differences  between  the  two 
curves  over  an  extended  range  of  cost  for  both  the  Aviation 
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and  the  Telecommunications  sponsors.  The  differences  be- 
tween the  curves  for  the  Intelligence  sponsor  fall  in  the 
middle. 


3.0  ADVANTAGES  OF  THE  COST-BENEFIT  APPROACH 


The  cost-benefit  methodology  developed  for  this  appli- 
cation should,  in  theory,  lead  to  better  resource  allocation 
decisions  if  valid  benefit  and  cost  information  is  used.  In 
this  application,  however,  there  was  considerable  (and 
understandable)  disquiet  on  the  part  of  USMC  sponsors  as  to 
the  validity  of  available  life-cycle  cost  data.  Consequently, 
the  final  POM  preparation  for  FY  1979  reflected  a benefit- 
only  approach.  Action  has  been  initiated,  however,  to 
develop  an  improved  cost  base  for  the  FY  80  POM  cycle  in  the 
expectation  that  better  cost  data  will  encourage  the  use  of 
the  cost-benefit  approach. 

3.1  Justification  of  POM  Recommendations 


The  systematic  assessment  process  that  generated  the 
benefits  and  produced  the  cost-benefit  ordering  of  procure- 
ments also  produced  discussions  and  information  that  supported 
and  justified  the  quantifications  of  benefit.  The  sponsors 
used  this  information  to  write  terse  descriptions  of  the 
supporting  rationale  for  the  benefit  numbers  assigned  to 
each  procurement.  These  descriptions  were  used  when  others 
asked  for  clarification  of  the  benefit  scale.  Additional 
attributes  of  this  process  are  its  responsiveness  to  requests 
for  "what-if"  analyses,  and  its  adaptability  to  changes  that 
may  occur.  The  cost-benefit  approach  itself  generates  part 
of  this  responsiveness  and  adaptability  because  the  theory 
dictates  how  changes  are  incorporated  and  what  the  resultant 
rankings  of  prospective  procurements  are. 

3 . 2  Availability  of  Interactive  Computer  Support 

An  IBM  5100  mini-computer  was  programmed  to  do  most  of 
the  calculations  and  data  storage,  retrieval,  and  manipulation 
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needed  by  the  working  group  responsible  for  preparation  of 
the  POM.  The  software  in  this  coinputer  was  interactive  in 
the  sense  that  the  officers  responsible  for  POM  preparation 
were  able  to  use  it  after  a very  short  instructional  period 
without  the  assistance  of  a computer  programmer.  They  could 
make  changes  to  the  data  and  ask  for  new  displays/  printouts 
at  their  own  convenience  without  relying  on  others  or  waiting 
in  the  queues  often  associated  with  large  computer  systems. 
Their  turn-around  time  was  on  the  order  of  minutes  or  hours, 
and  they  could  take  the  computer  to  meetings  and  briefings 
with  them.  This  gave  the  officers  a high  level  of  confidence 
in  the  output  of  the  computer  because  they  were  controlling 
the  inputs  and  the  computer  processing  themselves. 

3.3  Identification  of  Critical  Decision  Areas 


The  cost-benefit  approach  also  facilitated  the  rapid 
identification  of  the  real  decision  points;  i.e. , that  sub- 
set of  procurements  in  the  "gray  area"  of  the  decision- 
making process  quickly  surfaced.  Consequently,  most  of  the 
subsequent  discussion  was  focused  on  the  troublesome  subset 
rather  than  spread  evenly  across  all  the  procurements . For 
instance,  there  were  a few  procurements  that  senior  officers 
felt  a priori  should  be  funded.  However,  when  the  final 
analysis  indicated  otherwise,  the  POM  working  group  was  able 
to  show  which  procurements  would  then  have  to  be  sacrificed 
to  fund  them.  In  all  cases,  this  type  of  comparison  was 
convincing. 

A major  decision  point  for  this  year's  POM  surfaced 
immediately  after  all  of  the  benefits  were  assessed  and 
entered  into  the  computer.  The  expected  TOA's  for  the  five- 
year  period  1979-1983  were  very  uneven,  with  a minimum  in 
1980  (Table  1-1).  The  1980  TOA  had  been  known  for  some 
time,  but  its  impact  was  not  clear.  However,  both  the  cost- 
benefit  and  benefit-only  orderings  indicated  that  there 
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was  a higher  rate  of  expenditure  of  funds  in  FY  1980  than  in 
any  other  fiscal  year.  Since  the  Marine  Corps  does  not 
have  a bank  available  for  smoothing  out  cash-flow  problems, 
this  was  serious.  In  fact,  a major  restructuring  of  the 
procurement  outlays  was  required  after  a short  analysis 
indicated  that  the  rearrangement  of  a few  procurements  would 
not  solve  the  problem.  The  sponsors  were  asked  to  provide 
several  alternative  funding  profiles  for  all  procurements 
requiring  1980  funding.  Short  descriptions  of  the  disad- 
vantages of  these  alternatives  were  also  provided,  including 
any  changes  in  benefit  numbers.  The  computer  software  was 
then  used  to  iterate  towards  the  solution  of  this  problem  in 
a short  period  of  time. 

3 . 4 Education  of  POM  Working  Group 

In  the  past,  there  have  been  sponsors  who  were  very 
displeased  with  the  final  POM  decisions  and  felt  that  their 
procurements  were  funded  at  a lower  level  than  optimum. 

Since  the  sponsors  were  uncertain  about  how  much  control 
they  had  over  the  outputs  of  the  cost-benefit  approach,  they 
were  initially  skeptical.  However,  not  only  were  all  the 
sponsors  satisfied  that  the  approach  had  been  useful  and 
educational  in  the  end,  but  those  who  were  in  a position  to 
feel  shortchanged  were  satisfied  that  they  had  been  treated 
fairly. 

Several  factors  contributed  to  the  sponsors'  satisfaction 
with  the  cost-benefit  approach.  First,  the  logic  and  recorded 
rationale  for  the  quantified  benefits  gave  people  confidence 
that  all  of  their  inputs  were  being  used.  Second,  the 
process  of  eliciting  benefits  generated  a substantial  amount 
of  discussion — discussion  that  resulted  in  the  education  of 
many  more  people  concerning  the  pros  and  cons  of  each  of  the 
procurements.  This  educational  process  involved  both  the 
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generalists  as  well  as  the  experts.  It  gave  the  sponsors 
more  understanding  of  each  others'  programs,  which,  in  turn, 
resulted  in  a greater  understanding  of  the  decisions  that 
were  made. 

The  Marine  Corps  programmers  responsible  for  putting 
the  POM  together  supervised  a committee  of  experts  (the 
sponsors)  to  complete  its  job.  Committees  are  necessary 
when  diverse  technical  information  must  be  digested  and 
summarized.  However,  it  is  often  difficult  to  focus  com- 
mittee debates  on  important  topics;  they  tend  to  ramble, 
resulting  in  a significant  waste  of  time.  By  using  the 
cost-benefit  approach,  the  supervisors  found  that  they  had 
much  more  control  than  usual  because  they  could  adjourn  the 
meetings  of  the  committees  when  the  discussion  began  rambling, 
analyze  several  alternatives  to  the  solution  of  the  committee's 
various  problems,  and  then  put  a recommendation  before  the 
committee  for  specific  comments.  This  expedience  was  only 
possible  because  of  the  logic  of  the  cost-benefit  approach 
and  the  group's  direct  access  to  the  computer. 

3.5  Facilitation  of  Future  POM  Efforts 


Finally,  next  year's  foundation  for  the  POM  is  far  more 
complete  as  a result  of  this  effort.  The  benefit  scales  for 
the  programmed  future  procurements  exist  although  changes 
resulting  from  new  information  and  new  programs  are  expected. 
The  rationale  documenting  these  benefit  scales  with  the  in- 
formation used  for  POM  79  also  exists.  Consequently,  it  is 
expected  that  the  elicitation  efforts  will  be  significantly 
reduced  for  development  of  POM  80. 
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4.0  CONCLUSIONS  AND  FUTURE  WORK  ; 

I 

i 

The  cost-benefit  approach  to  ranking  procurements  for  j 

future  capability  has  demonstrated  utility.  It  helped  to  i 

make  better  decisions,  to  provide  a controlling  structure  j 

for  the  decision-making  process,  and  to  clarify  reasons  for  I 

the  decisions  so  that  programs  could  be  fully  justified  in  j 

the  federal  review  process.  | 

i 

One  valuable  extension  of  this  work  would  be  to  apply  | 

the  cost-benefit  approach  to  the  entire  Marine  Corps  program, 
that  is,  to  the  four  types  of  funding — manpower,  operations 
and  maintenance,  reserves,  and  procurement.  This  would  not 
only  systematize  and  improve  the  decision-making  processes 
in  these  areas,  but  would  enable  attention  to  be  focused  on 
how  the  last  dollars  are  being  spent  in  each  appropriation. 

This  scrutiny  would  facilitate  the  comparison  of  the  relative 
benefits  of  these  four  separate  allocations  to  determine 
whether  it  would  be  advantageous  to  reallocate  funds  among 
them.  That  is,  they  would  have  the  detailed  logic  necessary 
to  decide  whether  too  much  money  is  allocated  to  procurement 
and  not  enough  to  the  operations  and  maintenance  areas. 

Another  extension  of  this  project  effort  would  be  the 
improvement  of  the  computer  software.  As  DDI's  application 
of  this  research  is  expanded  and  improved,  better  and  more 
flexible  software  will  be  needed  to  increase  the  current 
level  of  efficiency  provided  by  this  new  analytic  approach. 

Although  an  integral  part  of  the  cost-benefit  approach 
is  the  estimation  of  life-cycle  costs,  this  was  not  emphasized 
in  the  POM  79  effort.  The  consensus  at  the  end  was  that  the 
cost  data  proved  to  be  the  weakest  link  of  the  cost-benefit 
analysis.  Therefore,  it  is  essential  that  both  a methodology 
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and  a data  base  be  developed  for  deriving  reliable  estimates 
of  life-cycle  costs. 

Finally,  zero-base  budgeting  is  now  being  implemented 
throughout  the  federal  government.  Until  now,  the  programming 
of  the  budget  (determination  of  items  to  be  included  in  the 
budget)  and  the  actual  preparation  of  the  budget  have  been 
somewhat  distinct.  Zero-base  budgeting  forces  a closer 
union  and  requires  that  the  cost-benefit  approach  be  used  in 
choosing  among  alternatives  and  determining  priorities.  It 
focuses  attention  on  incremental  packages  of  these  alter- 
natives. This  is  exactly  the  approach  used  in  determining 
the  procurement  of  future  capability  items  for  POM  79;  the 
methodology  is  a logical  instrument  for  implementation  of 
zero-base  budgeting. 
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